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ABSTRACT 



This thesis addresses the benefits that the employees of the Naval Avionics Center 
(NAC) believe they are receiving from management training. Specifically, this study 
determines whether or not the management training that is offered to NAC scientists and 
engineers is perceived as beneficial. In order to determine the perceptions of the trainees, 
a questionnaire was developed and distributed to all eligible employees. Fifty-nine percent 
of the participants responded. The results of the research indicated that the trainees 
generally had positive responses to all of the survey questions. Based on those results, it 
is concluded that the scientists and engineers do perceive the training that they are 
receiving as beneficial. However, due to the small number of surveys returned, the results 
for the entire study must be considered exploratory. 
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BACKGROUND 



I . 

A. NAVAL AVIONICS CENTER 

The Naval Avionics Center (NAC) , which is located in 
Indianapolis Indiana, is a non-profit government organization 
that specializes in the field of aviation electronics, or 
"avionics." They cater not only to the specific needs of the 
four American services- -the Navy, Marines, Air Force, and 
Army--but to international customers as well. Their work 
force consists of over 3400 employees, 34% of which are 
scientists and engineers. The Center is organized into nine 
departments. (A basic organization chart is provided as 
Appendix A.) Four of these departments--department "200" 
(Manufacturing Technology), "400" (Product Integrity and 
Assurance), "800" (Systems and Technology), and "900” 
(Engineering) --are where a majority of the scientists and 
engineers are located. As civil servants, they are salaried 
employees who are paid on standard government GS/GM pay 
scales . 

NAC' s mission is "To conduct research, development, 
engineering, material acquisition, pilot and limited 
manufacturing, technical evaluation, depot maintenance, and 
integrated logistic support on assigned airborne electronics 
(avionics) , missile, spaceborne, under sea and surface weapon 
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[Ref. 1] 



With a 



systems and related equipment.' 1 
mission statement as diverse and technically oriented as 
NAC's, management training becomes an important issue in the 
development of their engineers. 

1 . NAC Management Training 

NAC management currently spends over ten million 
dollars annually to offer a variety of training opportunities 
for their employees . A large portion of the training budget 
is allocated to management training as can be attested to by 
the vast number of management courses available to NAC 
personnel (see Appendix B) . During the past couple of years,, 
NAC has been placing increasing emphasis on Deming training, 
otherwise known as Total Quality Management. The acceptance 
of this philosophy solidifies NAC's commitment toward training 
and to the continual learning of their employees. 

Instead of NAC employees taking courses piecemeal 
and without any guidance, NAC management has developed a 
program called the "Individual Development Plan" (IDP) . The 
IDP is a program that identifies, through both long and short 
term goals, the employee's desires for training development. 
If the employee elects to take part in the educational 
opportunities offered by NAC, then the IDP becomes an annual 
process by which the employee and the supervisor sit down and 
try to determine what type of specific training courses would 
be best for the employee to take. It is up to the individual 
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employee to decide which general route will be followed--! . e . , 
technical or managerial- Once that decision has been made, 
the employee and the supervisor meet and try to develop a plan 
that best fits his/her individual needs. 

NAC' s current management training program 
encompasses supervisory training development, management 
development and executive development. These broad categories 
can be broken down further to include such generic courses as 
General Management Skills and Managerial Aspects of 
Supervision, or courses that are designed to teach more 
specific skills such as interviewing, negotiating, or 
appraising performance . 

2 . Corporate Vision 

NAC realizes that its sole purpose for existence is to 
support the Fleet. In order to maximize the ef fectiveness of 
that support NAC has identified its corporate vision: "to be 
recognized as the leader in avionics and manufacturing 
excellence." [Ref. 1] To help achieve the corporate vision, 
The following excerpt from the Center' s overview statement 
describes the importance NAC management places on the 
continuing education and training of its scientists and 
engineers : 

. the center invests in a strong personnel training 
program designed to foster technical and managerial skills 
especially attuned to addressing the Navy's airborne 
electronics issues of today and tomorrow. In order to 
stay abreast of new philosophies in the systems 
acquisition process and the rapid advances in avionics 
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technologies, the Center continually invests in the 
upgrading of its personnel's capabilities. [Ref. 1] 

With this type of philosophy toward training and with the 

training that is currently being offered, NAC is taking 

positive steps toward the realization of that vision. 

B. PURPOSE AND SCOPE 

NAC management is committed to the further development and 
improvement of their employees. This is demonstrated through 
their $10M annual training budget and their continual interest 
in providing their employees with ample training 
opportunities. The scientists and engineers at NAC are 
offered training to further their technical abilities as well 
as develop their managerial skills. For the purposes of this 
study, the focus is limited to management training. NAC 
offers a wide variety of management training courses to their 
scientists and engineers. This study evaluates the 

effectiveness of that training from the perspective of the 
training participants, i.e., the managers or potential 
managers. TT A manager," as defined by R. W. Griffin, "is 
someone who plans and makes decisions, organizes, leads, and 
controls human, financial, physical, and information 
resources, with the aim of achieving organizational goals, in 
an efficient and effective manner." [Ref. 2] How does 
a person acquire the skills necessary to become a successful 
manager? The most common path is through a combination of 
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experience and proper management training. This training 
should not only provide for the accomplishment of managers' 
tasks as outlined above, but it should also be relevant to 
their jobs, as well as beneficial to themselves and their 
organization. This study addresses the benefits the people 
believe they are receiving from management training. 
Specifically, this study determines whether or not the 
management training that is offered by NAC to scientists and 
engineers is perceived as beneficial. 

Recently, NAC management requested that the Naval 
Postgraduate School conduct an evaluation of the return on 
investment (ROI) of NAC training and this study represents a 
part of that overall effort. Evaluating management courses is 
difficult at best. As a result, this effort utilizes 
perceptions of trainees to assess training outcomes, which can 
then be reviewed by NAC management to make judgements about 
the payoff of the training program for the organization. 

This study does not address specific cost benefits as that 
is the purpose of another of the efforts in this program of 
research. Rather, the issue of interest is the perceived 
outcomes of specific courses. 

C. RESEARCH QUESTIONS 

The primary research objective addressed in this thesis is 
whether or not specific management courses are perceived as 
useful by NAC scientists and engineers. To accomplish the 
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research objective, the following specific research questions 
about training outcomes are addressed: 

1 . How relevant is management training to job 

requirements according to trainees? 

2. How satisfied are trainees with specific management 
courses? 

3. What is the perceived relationship between management 
training received and changes in job performance? 

4. What are the potential benefits of the management 
training for the NAC organization? 

The assessments can be used in conjunction with other studies 
to provide NAC management with a basis on which to evaluate 
its return on investment for training. 

D. ORGANIZATION OF THE THESIS 

This thesis is divided into five chapters and three 
appendices. This chapter is followed by Chapter II, which is 
a review of the literature that is pertinent to the research. 
Chapter III provides the data base for this research and 
discusses the method used for the development and 
administration of the questionnaire. Chapters IV and V 
provide the results and conclusions, respectively, from the 
analysis of the questionnaire data. Chapter V also includes 
recommendations for NAC management training. The Appendices 
consist of NAC' s organizational chart, the list of management 
courses available, and the questionnaire used for this study. 
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II . 



LITERATURE REVIEW 



A. MANAGEMENT TRAINING 
1 . Introduction 

"No resource is more critical to the success of an 
organization than management talent. And no resource is more 
scarce. Effective managers are made, not born." 
[Ref. 3] For effective managers to be "made," some 
form of training must be implemented. William Thomas, author 
of "Training and Development Do Make Better Managers," 
[Ref. 4] says that: 

"The success of any kind of education or training is 
difficult to measure. Companies that invest time and 
money in developing their managers often see improvements 
in overall company performance. However, unlike 
manufactured products, good managers cannot be mass 
produced. It is the combination of personality, instinct, 
on-the-job experience, good mentoring, solid educational 
background, decision-making ability, and luck that makes 
a good manager." 

Thomas goes on to say that most successful large companies, 
companies like IBM, Walt Disney Productions, GE, and Dana 
Corporation, provide their managers with "educational 
luxuries." These luxuries include job rotation, succession 
planning, individual coaching, and, usually, sophisticated in- 
house training programs. Peters and Waterman in their book, 
"In Search of Excellence," identified 71 companies that they 
felt were the most successful in the country. Thomas 
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conducted a study of those companies and found that 97% have 
their own in-house management development programs, 96% pay 
for their managers to take courses in management from local 
colleges and universities, and almost half of those companies, 
43%, grant time off for study leave. [Ref. 4, p. 53] 

Through his study, Thomas further concluded that of 
those companies offering management development courses, 94% 
provided their own training staffs. By using their own 
personnel as instructors, the company's "corporate culture" 
can be transmitted to the managers and supervisors of the 
future. [Ref. 4, p. 53] Corporate culture is an important 
element to convey because it instills within the recipient a 
sense of pride and commitment to the organization. 

Lastly, Thomas states that "Management education is 
cost effective only if you think it is. Its payoff lies in 
the future- It is an investment in the capital resource of 
leadership, of providing educational breadth to enable a 
person to achieve managerial depth." [Ref. 4, p. 53] It is 
hoped that through the investment in training, successful 
managers are made. As stated previously though, being 
successful is the combination of personality, instinct, on- 
the-job experience, good mentoring, solid educational 
background, decision-making ability, and luck. However, if a 
continuing and sound educational/training program is 
implemented it can go a long way toward making successful 
managers . 
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2 . Management Training in The Workplace 
a. In-House Training 

With jobs in today's workforce becoming 
increasingly more complex, there is greater concern over 
whether the education and training received by the workers 
will fulfill the requirements needed for them to function 
successfully in their chosen fields. According to Donald 
Kirkpatrick, an organization can improve its supervisors' 
performance through proper training. He states that there are 
three major in-house approaches for training and developing 
supervisors: selected reading, on-the-job coaching, and 
classroom programs [Ref. 5] . 

(1) Selected Reading 

This is probably the most difficult of the 
three to accomplish because encouraging supervisors to learn 
through reading is not an easy task. It requires books or 
articles that are easy to read and readily accessible, but 
there must be some motivational factor built into the reading 
program. [Ref. 5, p. 52] The reading material should cover 
topics such as leadership, motivation, communication, decision 
making and problem solving, time management, and subordinate 
training. To stimulate participation and to motivate 
supervisors to read, the training/personnel department, as 
well as higher-level managers should also become involved in 
the program. This involvement can consist of (1) asking 
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supervisors to read certain books , then leading them in a book 
discussion; (2) discussing what was read in the book and how 
it could be applied to the workforce; and (3) having 
supervisors read chapters of a book and present a summary to 
each other. [Ref. 5, p. 52] 

(2) On-The-Job Coaching 

"On the job training takes place all the time: 
Managers are always training subordinates by setting examples. 
Training by example is one of the most powerful training 
techniques; coaching reinforces it." [Ref. 5 p. 52] 

In their every day job, managers coach 
supervisors in a number of ways: (1) by correcting mistakes, 

(2) through positive reinforcement, and (3) by completing the 
employees formal performance appraisal. To become more 
effective at appraising an employee's performance or 
conducting proper coaching procedures, Kirkpatrick recommends 
the following steps: 

* Clarify what is expected of the employee. 

* Tell the employee how he/she is doing. 

“ Select one or more performance areas in which the employee 
needs to improve. 

* Develop a performance improvement plan. 

* Coach to implement the plan. 

Managers should be schooled in the proper principles and 
techniques of coaching in order to get the most out of their 
employees . 
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(3) Classroom Programs 

If a company is large enough, classroom 
programs should be conducted in-house. The term "in-house” 
doesn't necessarily mean that the program takes place on the 
company premises. It simply means that attendance is 
restricted to managers from one organization. The main 
advantage of in-house programs is that the course or courses 
being taught can be structured to meet the specific needs of 
the organization. Parry and Robinson [Ref. 3, pp . 8-13] 

describe classroom programs that are run by outside 
organizations as taking the Johnny Appleseed approach. "If 
you scatter enough seed on enough ground, some will take root 
and produce." Programs are offered by outside organizations 
such as colleges and universities, the American Management 
Association, and trade associations. In creating a good in- 
house training program, Kirkpatrick identified three phases 
that must be followed: planning, conducting, and evaluating. 

[Ref. 5 p. 54] See Table 1 for an outline of the 

characteristics that help to make a good in-house training 
program. In-house programs can develop, within the employee, 
the attitudes that the organization wants. If set up 
properly, the supervisor's knowledge will increase and his or 
her skills will improve. 

This same point is made again by Anthony P. 
Carnevale. Management training for middle managers is common 
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TABLE 1 



GOOD IN-HOUSE TRAINING PROGRAMS 



PLANNING 


Obj^oti-r* 


Subjaat Content 


3ah*dul* 


Location 


Based on needs 
Stated in terms of 
expected outcomes 
Learner-oriented 
Clear to everyone 


Based on objectives 
Understandable to 
learner 

Pertinent to learner 
Of benefit to learner 


Convenient to learner 
Proper length 
Proper frequency 


Adequate facilities 

Minimum 

interruptions 




CONDUCTING 




L^ad^r* 


Method* 


Aids 


Knowledgeable in subject 
Interested in teaching 
Respected by learners 
Skilled in commun icat ing 
Skilled in teaching 


Varied 

Maintain learner interest 
Get group involvement 
Help learners learn 


Audio 

Video 

Simple 

Minimum cost 

Help learners learn 


EVALUATING 


Reaction 


Learning 


Behavior 


Results 


How well did 
learners like the 
course? 


To what extent did 
participants learn 
facts, principles, 
skills and 
attitudes? 


To what extent did 
job behavior change 
because of the 
course? 


What final results 
were achieved? 



in American companies. Although new managers usually start 
with a good educational base, they often have little training 
in motivating and managing people. Consequently, as they move 
from their areas of technical expertise and begin to take on 
the responsibilities of coordinating resources and people, 
they usually find that their managerial skills are lacking. 
To alleviate this problem, American employers try to "make" 
managers through an extensive employer-provided training 
programs, including on-the-job training and mentoring. 
[Ref. 6] 
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Jb . Costs /Effects of Tra.xni.ng 

In a 1984 survey focusing on formal employer- 
sponsored training, Carnevale and Garner estimated that 
employers spent approximately $30 billion in direct costs for 
formal training during that same year. [Ref. 7] This 

is consistent with another 1984 survey in which similar 
figures were obtained. [Ref. 8] 

Barron and Black took the above study and went one 
step further. Instead of looking at training costs, they 
wanted to take a look at the effects of training on 

employment, i.e., productivity and wages. What they concluded 
was that a 10 percent increase in the amount of time devoted 
to training was associated with an average 3 percent increase 
in productivity and a 1.5 percent increase in wages. 

[Ref. 9] 

After evaluating the above mentioned studies, Gary 
Becker, in his article "What We Do and Don't Know About 

Training in the Workplace," [Ref. 10] concluded that 
there are some issues concerning training that are consistent 
from one company to another. These consistencies include: 

- The likelihood of receiving training declines with age. 

* The likelihood of receiving training increases with 
education . 

* Men are more likely to receive training. 

* Most training is informal. 
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In addition to finding out what we know about training, Becker 
also found out what we do not know: 

* We have no universal definition of training. 

* It is unclear how to measure whether training is 
successful . 

* Changes in training over time are unknown. 

3 . The Future of Management Training 

a . Increased Management Personnel 

M In the future there will be more management and 
more managers.” [Ref. 6, p. 21] Anthony Carnevale believes 
this because, as seen over the past decade, a majority of the 
jobs in today's organizations that require less skill have 
been replaced by jobs using sophisticated technology, which 
require more highly skilled employees. According to the 
Bureau of Labor Statistics, between now and the year 2000 the 
number of managers, professionals, and marketing personnel 
will increase by more than 30 percent. The number of 
management jobs is expected to increase by three million, more 
than one and a half times the rate of increase in other 
occupations. [Ref. 11] The question is this, will 

management training have to change in order to keep pace with 
the increasing changes in technology? 

b . Changes in Training 

David Burgoyne, author of "Management Development 
Programs: Then and Now," answers the above question in the 

negative. According to Burgoyne, "Although the changes 
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businesses experience in the next few decades may be massive, 
the role of management development programs will be a static 
one: to prepare managers to handle change and challenges more 
effect ively - " [Ref. 12] 

One of the best places for a new manager to learn 
how to perform many of the necessary management skills is on 
the job. Skills such as leadership, communication, 
interpersonal effectiveness, teamwork, negotiation, learning, 
counseling, and teaching are difficult to teach in school but 
through applied learning, are easy to teach on the job. This 
does not mean however, that we should leave the development of 
new managers to the accidents of experience. American 
companies are better at providing applied learning than the 
schools. The first challenge employers must face is the 
difficult task of teaching the schools a more applied approach 
to management education and to encourage the schools to add 
curriculum in the "soft" skill areas. It will not be enough 
for the organizations to duplicate the academic curriculums, 
they must add the applied dimension to their training to be 
effective. [Ref. 6, p. 28] 

Hawkins and Barclay describe the changes in 
technology as "extremely complex, inter-disciplinary based and 
constantly evolving, through radical and refining changes, in 
response to the needs of the business." [Ref. 13] 
These changes in technology have changed the nature of 
management within manufacturing companies. Hawkins and 
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Barclay refer to these changes as the old and new orders of 
management. Table 2 summarizes the differences between the 
"Old and New" orders. 



TABLE 2 

OLD AND NEW ORDERS OF MANAGEMENT 



Old Order 


New Order 


Production oriented 
Efficiency based 
Authoritarian 
Stable structures 
Functionally based 


Market oriented 
Entrepreneur based 
" Co - ope rat ive " 
Changing structures 
Inter-disciplinary 
teams 



As can be seen from the table, the main difference between the 
two is that the new order of management is focused more toward 
controlling risk and increasing responsiveness to the 
customer . 

In addition to these differences in management, 
engineers must change also. Engineers of the future must be 
"commercial engineers" with skills such as: contract 
negotiating and closing, selling and marketing, and persuasion 
and leadership qualities. The Finniston Report 
[Ref. 14], agrees with Hawkins and Barclay in 
stressing the need for future engineers to acquire and use 
these commercial skills. In order for tomorrow's engineers to 
become proficient under the "new order" style of management, 
it is essential that they learn and practice the commercially 
based skills. In a survey conducted by Hawkins and Barclay 
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[Ref. 13, p. 50], employers were asked what qualities were 
considered most important when promoting engineering graduates 
to management positions. The responses are shown in Table 3. 



TABLE 3 

FACTORS CONSIDERED FOR MANAGEMENT PROMOTION 



Factor 


Percentage 

Rating 


People Management 


95 


Communication Skills 


90 


Track Record 


89 


Interpersonal Skills 


87 


Technical Awareness 


85 


Project Leadership 


84 


Personality 


83 


Prof it /Efficiency 


80 


Innovative Skills 


77 


Training Undergone 


68 



As can be seen from the table, people-related skills are 
considered to be among the most important, whereas 
innovative type skills were lower on the list. 

Another factor to be considered when implementing 
training is the importance of timing. If management training 
is given too early in an individual's career it will be 
difficult for them to understand some of the concepts and 
ideas. On the other hand, if management training is given too 
late, when the individual already has a large staff, there is 
little chance to experiment in developing one's own management 
style. The balance between being there long enough and not 
being there too long must be found. According to Hawkins and 
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Barclay,- this balance can be found "when the individual is in 
charge of a small number of people." It is at this time that 
the new managers are more peer oriented and have more support 
from their peers , allowing them to experiment with the new 
knowledge, skills, attitudes, and behavior that they've 
attained. [Ref 13, p. 50] 

The last point to be brought up about management 
training is that in the year 2000 it is estimated that, of a 
company's current workforce, 70 percent of that workforce will 
be employed by the same company for whom they currently work. 
This implies that in order to ensure adequate numbers of 
potential managers, attention must be given to developing the 
existing personnel. [Ref. 13, p. 54] 

4 . SUMMARY 

This section has provided an overview on management 
training. The first subject referred to was in-house 
training. It was determined that organizations, through 
proper in-house training, can develop within their employees 
the desired attitudes, culture, and techniques needed to 
accomplish their routine day-to-day tasks. However, for the 
in-house training program to be effective, three phases should 
be followed: planning, conducting, and evaluating. In-house 
training, according to Donald Kirkpatrick, can consist of: 
reading programs, on-the-job coaching, and classroom programs. 
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The second subject referred to was the future of 
management training. It was pointed out that management 
development programs are moving away from the 
technical/innovative aspect of training and toward a more 
people-oriented type of training. Until the schools can be 
taught to teach these kind of "soft" skills, more importance 
will be placed on in-house training. 

With more importance being given to in-house training, 
more attention will have to be given toward evaluating those 
courses . 

B. E VALUAT I ON OF TRAINING 
1 . Evaluation Measures 

Training directors need to know if the employees are 
learning from the training they are receiving. If they are 
learning, the training directors need to know if the employees 
are applying it to their jobs. The bottom line is this; is 
the organization receiving an equitable return on its training 
investment? To find out, training or development programs 
should be evaluated. There are two basic kinds of evaluation 
measures: those collected in or at the end of training and 
actual performance measures collected when the trainee has 
returned to the job. The former are easier to get but the 
latter are more valid. Trainees may say that they enjoyed the 
training and learned a lot, but the true test is whether their 
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job performance is better after their training than before. 
[Ref. 1, p. 364] 

2 . Criteria used to Measure Training 

The goal of training evaluation is to determine the 
degree to which the basic relationship between training and 
job performance is accomplished. In order to determine 
whether or not training was successful, Kirkpatrick identified 
four levels of criteria that should be used when evaluating 
supervisory training programs. They are: reaction, learning, 
behavior, and results. [Ref. 15] The four criteria 

will be discussed below to help clarify the meaning of each, 
their relationship with each other, and the guidelines and 
procedures for using them, 
a. Reaction 

Reaction can be defined as how well the trainees 
liked a particular training program. Evaluation in terms of 
reaction is the same as measuring the feelings of the 
participants. It is simply a measurement in determining how 
satisfied the participants are with the training they have 
received. When measuring a participant's reaction to 
training, honest and meaningful answers are desired. To 
obtain candid responses, the following five guidelines should 
be followed: 

1. Determine what you want to find out. 
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2 . Use a written comment sheet covering those items 
determined in step one. 

3. Design the form so that the reactions can be tabulated 
and quantified. 

4. Encourage the participants to write in additional 
comments not covered by the questions designed in step three. 

5. Obtain honest reactions by making the forms anonymous . 

Even after following the above guidelines, there is 
still no assurance that any learning has taken place because 
measuring reaction does not include a measurement of learning. 

Jb . Learning 

Learning is defined as the extent to which the 
trainees have learned the information and skills presented 
during the training course. To establish a procedure for 
measuring the amount of learning that takes place in a course 
of instruction, several guidelines should be followed: 

1. A before-and-after test should be used to ensure that 
any learning is a result of the program. 

2. The learning should be measured on an objective basis so 
that quantitative results can be determined. 

3. Where practical, a control group, a group not receiving 
the training, should be compared with the experimental group 
that receives the training. 

These guidelines imply that evaluation in terms of 
learning is much more difficult than evaluation in terms of 
reaction . 
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c. Behavior 



Behavior is basically the extent to which on-the- 
job behavior of the participant has changed as a result of the 
training received. Robert Katz, a Professor at Dartmouth, 
wrote an article in 1956 for the Harvard Business Review 
entitled, "Human Relation Skills Can Be Sharpened." 
Kirkpatrick [Ref. 16] quotes Katz as saying, "If a 
person is going to change their job behavior, five basic 
requirements must exist": 1) They must want to improve, 2) 

they must recognize their own weaknesses, 3) they must work 
in a permissive climate, and 4) they must have an opportunity 
to try out the new ideas. Without these four requirements 
being met, it will be difficult, if not impossible for the 
participants to change their job behavior. 

How do we prove that their behavior has changed? 
Kirkpatrick identifies four guidelines that should help: 

1. Measure behavior on a before-the-program and after-the- 
program basis. 

2 . Allow enough time between the program and the after-the- 
program evaluation to allow for change in behavior to occur. 

3. Get information about the participant from as many 
sources as possible. 

4. If practical, use a control group to compare with the 
experimental group. This will help to prove that any changes 
in behavior were due to the program and not to other factors 
such as salary adjustments, coaching from the boss, or 
influence from other people inside or outside the 
organization . 
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There will be some difficulties associated with the 



above guidelines. First of all, measuring behavior before the 
program, step one, is a difficult task. When measuring 
behavior before the program, the measurement would have to be 
taken enough times to be sure that it was not only accurate, 
but that the behavior observed was typical as well. Secondly, 
concerning step two, how much time between the program and the 
evaluation is enough time? The time may vary for different 
people and different types of skills so it may have to be 
measured several times to be sure that the measurement is 
accurate. As can be seen, evaluating training programs in 
terms of on-the-job behavior will be more difficult than 
measuring either reaction or learning. 
d . Results 

Results are the outcome of the training program. 
"It would include such things as improved productivity, better 
quality, lower costs, meeting deadlines, more competition, 
reduced accidents, improved morale, lower turnover, and 
ultimately, more profits or better service." [Ref. 15, p. 6] 
Determining what final results were accomplished due to the 
training program is the most difficult part of evaluating 
results. There are so many other factors involved that it is 
frequently impossible to prove that the training program 
caused the desired result. Again, as with the three previous 
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criteria, Kirkpatrick has developed the following guidelines 
to help in evaluating results: 

1. Measure the conditions before the program and compare 
with the conditions after the program. 

2. Use control groups to try to eliminate other factors 
that could have caused changes in the results. 

As we look at the evaluation of reaction, learning, 
behavior, and results, we can see that it is much easier to 
say that the training program is having a positive effect on 
the above criteria than it is to prove it. In some cases, 
proof is impractical and almost impossible to get. If this is 
the case, what do you do? According to Kirkpatrick, ?, you 
shoot for the proof but be satisfied with the evidence.” 
[Ref. 17] 

Jackson and Kulp [Ref. 18] suggest that it 
is better to approximate evaluation than neglect it all 
together due to its difficulties. For example, if you can't 
get measures of behaviors and results, then measuring 
learning, both at the end of the course as well as some time 
later, is better than doing nothing at all. 

3 . Evaluation Techniques 

In his article, ”A Training-Appropriations Process”, 
David L. Hobbs identifies two evaluation techniques: 
interviews and surveys. [Ref. 19] According to 

Hobbs, "An advantage of interviews and surveys is their focus 
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on the individual. Respondents are not reacting to or 
influenced by the opinions of others. They are more free to 
express their own ideas and are more candid and specific in 
their comments. The interviewer or surveyor benefits from 
having a high level of control over the direction and depth of 
the inquiry.” [Ref. 19, p. 110] Table 4 identifies 
additional advantages as well as some disadvantages associated 
with administering a questionnaire. [Ref. 18, p. 28] 



TABLE 4 

ADVANTAGES AND DISADVANTAGES 
OF QUESTIONNAIRES 



Advantages 


Disadvantages 


1 . Can usually be 
administered to a large number 
of people, with less time, and 
money than most other 
strategies . 


1. Can obtain only estimates 
of what people actually think, 
feel, do, or produce. 


2. Ensures that everyone 

responds to identical 
questions/ directions . 


2. Responses may be biased be 
respondents' desire to present 
themselves in the best 
possible light-or to respond 
in a way that they think will 
please the author or recipient 
of the questionnaire. 



When using a questionnaire, you must ensure that it is 
valid and reliable. If the questionnaire measures what it is 
supposed to measure then it is considered valid and if it 
measures consistently then it is reliable. To determine if a 
questionnaire is valid to the area of study, its questions are 
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compared with the program objectives. If each objective is 
covered by at least one question, the questionnaire meets the 
basic criterion of validity. [Ref. 20] 

4 . Contingency Variables 

Instead of measuring variables before and after the 
training course or using control groups to see if any 
improvement occurred as a result of the training, Clement and 
Aranda decided to look at variables beyond the training 
course- [Ref. 21] These variables, referred to as 
"contingency" variables, include the organizational setting 
within which the manager attempts to use the training, the 
nature of the manager to be trained, and the nature of the 
problem to be solved through training. 

a. Organizational Influence 

Robert House [Ref. 22] identified three 
factors that affect the transfer of training from the 
classroom to the job: the formal authority system within the 
organization; the immediate superior of the trainee; and the 
primary work group of the trainee. The first factor refers to 
the possibility that top management has established company 
policies and objectives. Any training that does not comply 
with those pre-existing standards will not succeed. The 
second factor, the supervisor, has a tremendous impact as to 
whether or not the trainee implements any of the new ideas. 
If the immediate supervisor does not agree with what the 
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trainee was taught, and that supervisor controls the rewards 
and punishments, then the trainee is not likely to use what he 
or she has learned. The last factor, the trainee's primary 
work group, refers to the possibility that the trainee may 
succumb to the expectations or pressures from his peers and 
subordinates , which may be inconsistent with training. 

All three of the factors listed above can influence 
a manager's attempt to use the concepts learned in a training 
course. To ensure that the factors have a positive effect on 
the trainee, all courses taught should reinforce the policies 
and practices of upper management and all personnel should be 
indoctrinated as to what upper management expects of them. 

Jb. The Nature of the Manager Being Trained 

All managers differ in their experience, education, 
abilities, attitudes, and values. Therefore, the training 
received by each individual, although the same, will have 
different effects on each person. To determine the change in 
supervisory attitudes, researchers Hayes and Williams, in a 
1971 study, found that attitude change is inversely related to 
age, seniority, and span of control. [Ref. 23] The 
study concluded that: "supervisory training programs are more 
effective when the participants are young, relatively new to 
supervisory ranks, responsible to a small number of 
subordinate personnel, and have a short period of total 
service . " 
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c. The Problem to be Solved Through Training 

This contingency variable is concerned with 
matching the proper training method to the desired training 
objective. A recent survey of training directors has found 
that case studies are best for the development of problem- 
solving skills; programmed instruction for retention of 
knowledge; and both role playing and sensitivity training for 
changing attitudes and improving interpersonal skills. 
[Ref. 24] The success or failure of a particular 
training program could rest with the type of training 
technique used. 

C. EVALUATION OF MANAGEMENT TRAINING 

Adequate evaluation of training in organizations is rarely 
done. [Ref. 25] Many articles have been written on 
the topic, but most address the difficulties inherent in the 
process. For example, negligence on the part of management in 
defining appropriate evaluation criteria, the difficulty of 
establishing scientifically matched control groups in field 
research settings, and that small numbers of trainees makes 
hypothesis testing and inferential statistical analysis 
difficult. Additionally, good evaluation is time consuming 
and expensive; and to compound the difficulties, training 
administrators cannot come to an agreement on a definition of 
"good” training evaluation. [Ref. 26] 
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The evaluation of management training has its own set of 
inherent problems- One of those is the difficulty in 
collecting data on "hard” criteria. This lack of "hard” 
criteria is not unusual in the world of management training 
where expected outcomes are often not stated or, when 
specified, are hard to measure. Jackson and Kulp have 
identified through their research many differences between 
management and technical performance that might influence 
evaluation. [Ref. 18, p. 5] These differences are: 

* The nature of management training makes it harder to 
describe, observe, and measure outcomes as compared to 
technical training . 

* Performance standards for management positions are less 
likely to be defined or agreed upon. 

- Management positions tend to deal more with qualitative 
problem-solving situations whereas technical problems tend 
to be more concrete. 

~ Management outputs are harder to judge for quality and 
often times they are not related to specific tasks. 

* Management performance often requires responding to 
unpredictable, unscheduled events. 

Jackson and Kulp have identified other differences between 
management and technical performance, however, they felt that 
those listed above were the reasons most likely to influence 
the evaluation of management training. 

D. SUMMARY OF LITERATURE REVIEW 

To summarize, the literature reviewed consisted of three 
main topics: management training, the evaluation of training, 
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and specific difficulties involved in evaluating management 
training. Within the first topic, it was determined that 
future managers will have to learn a new style of management-- 
away from the production oriented, authoritarian style--toward 
a more market oriented, cooperative style. Additionally, 
engineers who want to become managers will have to be more 
people oriented vice technically innovative. To help 
accomplish this, in-house training should be used. In-house 
programs allow the organization to develop within the 
employees the attitudes and values that it feels are most 
important . 

Within the second topic, the evaluation of training, four 
criteria were developed to help measure the effectiveness of 
training. The four criteria used were: reaction, learning, 
behavior, and results. These four variables are incorporated 
before and after evaluations, as well as with control groups, 
to help measure whether or not the course was successful. 
Although it is difficult to meet these standards to prove what 
is successful because of training and what is successful 
because of outside circumstances, it is still better to 
attempt to determine the effectiveness of a training course 
than to do nothing at all. 

Finally, it was determined that management training is not 
evaluated as often as technical or clerical training. Of the 
several reasons listed above, the overriding explanation for 
this is the fact that most management training courses do not 
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state their training objectives. Those organizations that do 
state their training objectives, however, do not state them is 
measurable terms. These two conditions, makes evaluation of 
management training difficult if not impossible. 
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III. METHODOLOGY 



A . GENERAL 

This chapter focuses on the method used in the development 
and execution of the questionnaire which was used as the 
primary data gathering instrument for this research. Specific 
data about NAC' s management training program were collected 
through the use of this questionnaire. Section B discusses 
the development of the questionnaire. This section describes 
the survey questions that were designed to determine whether 
or not the individual's job performance is different as a 
result of the management training received. Section C 
discusses how the survey sample was determined. Section D 
discusses how the data were collected. And lastly, Section E 
provides recommendations pertaining to this and subsequent 
questionnaire development . 

B. DEVELOPMENT OF THE QUESTIONNAIRE 

As identified in Chapter II, there are four criteria used 
to measure management training: reaction, learning, behavior, 
and results. [Ref. 18] Due to the time constraints of this 
study, the only criterion we were able to evaluate was 
reaction. This was accomplished through the use of a 
questionnaire. The purpose of the questionnaire was fourfold: 
(1) to determine the relationship between management training 



32 



and job requirements, (2) to determine the satisfaction of 
trainees with specific management courses, (3) to determine 
opinions on the relationship between management training 
received and changes in job performance, and (4) to determine 
if it is perceived that MAC benefits from the management 
training the trainees receive. 

The quest ionnaire consists of a combination of closed and 
open-ended questions. However, to get the best response to 
the questions, it was determined that the majority of the 
questions should be open-ended vice closed. Open-ended 
questions allow respondents to answer freely in their own 
words and they can be worded in such a w ay that they either 
ask the respondent a broad general question or a relatively 
specific question. Table 5 identifies some additional 
advantages as well as some disadvantages of administering 
open-ended style questions. [Ref. 18 p. 22] 

Closed questions, on the other hand, are best used for 
demographic type questions and questions that don't require 
a specific explanation. Closed questions present respondents 
with possible responses from which they choose the most 
appropriate one(s) . This type of question can include a wide 
variety of responses, including multiple choice, matching, 
rating, ranking, true-false, or yes-no. Table 6 identifies 
both the advantages and disadvantages to using closed 
questions. [Ref. 18, p. 22] 
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TABLE 5 



ADVANTAGES AND DISADVANTAGES 
OF OPEN-ENDED QUESTIONS 



Advantages 


Disadvantages 


1 . Questions can usually be 
developed easily and quickly. 


1. Questions may be difficult 
to score and analyze because 
of the variability of the 
responses . 


2. Questions allow a free 

expression and may pick up 
information that would be 
missed in a closed question. 


2. Questions usually take 
more time and effort to 
complete than closed; result 
may be lower response rate if 
respondents can choose not to 
respond-or in brief, 
uninformative responses where 
there is no perceived choice. 



TABLE 6 

ADVANTAGES AND DISADVANTAGES 
OF CLOSED QUESTIONS 



Advantages 


Disadvantages 


1 . Can usually be scored and 
the results analyzed quickly 
and easily. 


1. Hay miss important 
information since they limit 
respondents' options. 


2. Can usually be completed 

quickly and easily and may 
result in a higher response 
rate . 



The questions asked on the questionnaire were designed 
after an initial trip was made to NAC in which 12 engineers 
from randomly selected departments were interviewed. The 
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information obtained from the interviewees enabled the 



researchers to develop a questionnaire that was custom 
designed for the scientists and engineers employed at NAC. 
The research questions , identified in Chapter I, can each be 
linked to one or two specific questions contained within the 
questionnaire. Research questions number 1 and 2 are answered 
by survey questions 7 and 8a, and 8d and 9 respectively; 
research question 3 is answered by survey questions 4, 5, and 
8b; and lastly, research question 4 is answered by survey 
questions 6 and 8c - See Appendix C for a copy of the 
questionnaire used in this study. The questions in the 
survey were specifically designed to help evaluate NAC' s 
management training program. 

Question 8 from Section II confused some of the 
respondents. The question asked respondents to rank, from 1- 
5, how they feel about certain statements. In this 
questionnaire, 1 stood for strongly disagree and 5 stood for 
strongly agree. After reading the respondents' answers to 
previous questions within the survey, it was determined that 

4 out of the 75 respondents reversed the ordinal ranking and 
gave, for example, a one when they really meant to give a 
five. This problem is considered minor because no more than 

5 percent of the respondents committed this error and no other 
consistent error patterns were noted. 
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C . SAMPLE 

NAC management identified a specific set of management 
courses that is offered primarily to its scientists and 
engineers . NAC feels that these 12 courses require special 
attention because, when compared to other management courses, 
these are not only more costly but they are attended more 
frequently as well. 

How was the determination made as to who would and who 
would not be sampled? In order to be chosen, the scientists 
and engineers had only to meet one criterion; they had to have 
taken at least one of the management training courses 
identified by NAC. This criterion, extracted from NAC's 
historical personnel data base, identified 239 individuals. 
However, because there is more than one research topic being 
conducted in this area of management training and training in 
general at NAC, the list of qualified participants had to be 
reduced. To eliminate "survey burnout, " 28 engineers were 
dropped from consideration because they had participated in a 
similar survey approximately two weeks earlier. An additional 
12 engineers were dropped because they had taken part in the 
initial interviews that determined what kind of questions 
needed to be included in the survey. Lastly, three more were 
excluded because two of the individuals had retired prior to 
administration of the questionnaire, and the third individual 
was erroneously identified as a qualified participant. In 
addition to the above 43 individuals, another 61 had to be 
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unexpectedly removed from consideration due to funding 
constraints for department 200. All eligible engineers from 
within that department had to be omitted. These reductions 
eventually pared the list of qualified engineers and 
scientists down to 135. 

D . DATA COLLECTION 

Initial plans called for all the survey participants to 
complete the questionnaire in a conference room on MAC 
premises. The participants were to be assigned into groups of 
30 and each group given 30 minute blocks of time to complete 
the form. However , due to financial constraints, MAC could 
not allow the 135 participants to take the time off to 
participate in the study. As a result, the data collection 
for the questionnaire consisted of the following 3teps : 

1. The researcher visited MAC for the actual 
administration of the survey. 

2. 135 questionnaires were given to the Branch 
secretaries for distribution. 

3. The participants were given two days to complete the 
quest ionnaires . 

4. At the end of the first day, only 13 questionnaires 
had been returned. 

5. During the second day, a reminder message was sent to 
each of the participants via electronic mail. A 
" received message” response was required. The message 
requested completion of the questionnaire by close of 
business . 

6. At the end of the second day, the researcher had only 
received an additional 29 questionnaires for a two day 
total of 42 . 
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7. The following week, another 38 questionnaires arrived 
in the mail, bringing the total up to 80 
questionnaires received . 

For reasons of anonymity, and in the hopes of obtaining candid 
responses, names were not requested on the questionnaires. As 
a result, not only was there no way to ensure completion of 
the questionnaires but there was no way to find out who did 
and who did not return them as well. The final response rate 
was only 59 percent (80 responses out of 135 surveys) . 

After analyzing the 80 returned questionnaires , three were 
rejected because the individuals had critiqued the wrong 
course and two more were rejected because the trainees did not 
attend the class. The analysis was performed with the 75 
remaining questionnaires; Chapter IV contains the results 
obtained from those questionnaires. 

E. RECOMMENDATIONS 

The recommendations provided here concern the distribution 
of the questionnaire and are not related to the conclusions 
and recommendations presented in Chapter V. 

Upon completion of the survey, one problem area was 
encountered pertaining to the distribution of the 
questionnaire. If at all possible, the individuals involved 
should be administered the questionnaire in a controlled 
environment. The definition for controlled environment in 
this case means two things: (1) the researcher should go to 
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the command/ organization where the questionnaire will be 
administered, and (2) the participants need to be gathered 
together in a single location; either in one large group or in 
many small groups. 

There are two reasons why the researcher should opt for a 
controlled environment. The first reason is to get a higher 
response rate. The simple fact is, a "captive" audience is 
more likely to respond to the questionnaire. Additionally, 
employees perceive the survey as important because work time 
is provided for the questionnaire to be completed. 
[Ref. 27] Secondly, the researcher of the 
questionnaire is available should any questions arise. This 
is important because it will ensure that any confusion about 
the questionnaire will be clarified and better survey data 
will be generated. 
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IV. RESULTS 



A . OVERVIEW 

The primary focus of this research is to determine: (1) 
if the trainees are receiving management training that is 
relevant to their job requirements; (2) if the trainees are 
satisfied with the specific management courses; (3) if there 
is a perceived relationship between the management training 
received and changes in job performance; and (4) if there are 
any benefits that NAC receives as a result of the management 
training. In answering these four points r this chapter 
presents a synopsis of the responses based on the data 
obtained from the questionnaires. These responses only 
concern the data received from the twelve management courses 
that NAC wanted examined. The remainder of this chapter 
consists of two sections. Section B is an analysis of the 
demographics of the scientists and engineers who responded to 
the questionnaire and Section C provides the results obtained 
from the survey data and conclusions. 

B . SURVEY RESPONDENTS 

As noted in the last chapter, the number of NAC employees 
who had taken the management courses of interest in the last 
year and a half was small. The number decreased even further 
due to the methodological difficulties described. As a 
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result , the validity of these results can be questioned. 
Under these circumstances , a characterization of the survey 
respondents is appropriate. By comparing respondents to the 
population of all who could have responded, and to the 
population of all NAC scientists and engineers, information is 
added about the representativeness of these data. These 
comparisons are made below. 

Table 7 identifies, by course, the percentage of surveys 
returned compared to how many people had actually taken the 
course. The purpose of Table 7 is to point out that the 
conclusions being drawn from the data may not be 
representative of the population of all those who have taken 
these courses . Although four of the twelve courses obtained 
a return rate of 50 percent or higher, they are still not 
practically significant because the initial number of eligible 
participants was quite small. Additionally, the first three 
courses, Basic Project Management, Dale Carnegie, and 
Performance Feedback all had a large number of eligible 
participants but the response rate to the questionnaire was 
very low. With these limitations, these data should be 
considered exploratory in nature. 

The data obtained from the questionnaire will provide to 
NAC a profile of the type of personnel who responded to the 
management training questionnaire. The specific areas of 
interest are: gender, years worked at NAC, paygrade, 
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TABLE 7 



PERCENTAGE OF PERSONNEL ELIGIBLE 
TO RESPOND COMPARED TO THOSE 
THAT ACTUALLY RESPONDED 



COURSE 


ELIGIBLE 
TO RESPOND 


ACTUALLY 

RESPONDED 


PERCENT 


Basic Project 
Management 


124 


38 


31 


Dale Carnegie Effective 
Speaking 


29 


10 


34 


Performance Feedback 


36 


6 


1* 


Power of Effective 
Listening 


14 


5 


36 


Train the Trainer 


7 


4 


57 


Managerial 

Communications Skills 


7 


4 


57 


Supervisory Institute 


25 


3 


12 


Dealing with Difficult 
People 


4 


1 


25 


Valuing Diversity 


14 


1 


10 


Assertiveness Skills 


7 


1 


14 


Teambuilding 


2 


1 


50 


Statistical Process 
Control 


1 


1 


100 



department currently working in, and lastly, supervisory 
experience . 
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The first area to be considered is gender. According to 
the data, 8 percent of the scientists and engineers who 
answered the questionnaire were females and 92 percent were 
males, within the organization as a whole, women represent 13 
percent of the scientists and engineers. 

As anticipated, the majority of the trainees who 
responded, 61 percent, have worked at NAC for ten years or 
less and 83 percent have not been formally designated as a 
supervisor. This makes sense because management training is 
intended mostly for those employees who are inexperienced in 
supervisory skills and relatively new to the organization. 
[Ref. 13] 

Related to the two demographics above is the third area of 
consideration--paygrade . There is a direct relationship 
between an individual's paygrade, number of years worked, and 
supervisory experience. In other words, if the individual is 
a GS-11, chances are that the trainee is not a supervisor and 
has only been at NAC for 10 years or less. The data show that 
74 percent of the trainees who participated in the study are 
between the grades of GS-9 and GS-12; within the NAC 
organization as a whole, 77 percent of the scientists and 
engineers are between those two grades. 

The distribution of research participants by department 
shows that departments 800 and 900, together, make up 
approximately 56 percent of the scientists and engineers 
within NAC. The next closest department, department 200, only 
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makes up 15 percent. The survey respondents from departments 
800 and 900 accounted for 81 percent of the study 
participants . 

C. ANALYSIS 

Since Basic Project Management and Dale Carnegie received 
response rates of over 30 percent, they will be analyzed in 
detail- The analysis of those courses addresses the four 
research questions identified at the beginning of this 
chapter. The remaining courses are analyzed as a group and 
will not specifically answer the four research questions. For 
both the detailed analysis and the group analysis, the 
numerical scores given to questions 8a-8d will be totaled 
individually by course. That total will then be divided by 
the number of respondents from each course and reported in the 
form of an average score. Question 8 was rated on a scale 
from 1-5 where one is "strongly disagree" and five is 
"strongly agree." An average score of five is considered a 
perfect score and corresponds directly to "strongly agree." 

In analyzing the open-ended questions--questions 4, 5, and 
6--a content analysis was conducted. As part of the analysis, 
the major themes, given by the respondents, were drawn from 
each question. The answers were then grouped according to 
theme . 
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1 . Objective One 

The first objective of this study is to determine if 
the trainees are receiving management training that is 
relevant to their job requirements. This question can be 
mapped to survey questions 7 and 8a. 

Question 7: Have you had any difficulty in applying 

anything you learned in the course to vour -job? If ves, why? 

Course: Basic Project Management 

N % 



a. No 22 58 

b . Yes 16 42 

c. Of yes responses: 

1) N/A to current job 8 50 

2) No time to implement 6 38 

3) Misc 2 12 



For this course, almost half of the respondents said that they 
did have difficulty in applying what they learned to their 
job. The two most frequently mentioned reasons were that the 
course was not applicable to their current job and that they 
were too busy to try the new process. 

Course: Dale Carnegie Effective Speaking 

N % 



a . No 8 80 

b. Yes 2 20 

c. Of yes responses: 

1) Difficult to implement 2 100 



Although this course has few responses, they do represent 34 
percent of the population that has taken the class. What is 
significant to note here is that 80 percent of the respondents 
indicated that they did not have trouble implementing what 
they learned to their jobs. 



45 



Question 8a: This course was relevant to my job, 

i.e-, it addressed problems and issues I encounter in my job. 



Course Avq . 

Basic Project Management (BPM) 3.92 

Dale Carnegie Effective Speaking (DC) 4.20 



The results show that the trainees thought that the DC course 
was more relevant to their jobs than the BPM course. 

The extreme numbers that make up the averages for the BPM and 
DC courses are: 3 ones and 13 fives for BPM and 1 one and 6 
fives for DC. It should be pointed out however that the one 
that was given to the DC course, in all likelihood, should 
have been a five. After reviewing all of this respondent's 
answers, it was felt that he reversed the scale for this 
question because the answers given were not consistent with 
the rest of his responses to the survey. With a five instead 
of a one, the average for this question increases to 4.60. 
This problem with the reversed scale will persist for the Dale 
Carnegie course throughout question number 8. 

2 . Objective Two 

The second objective of the study is to try to 
determine if the trainees are satisfied with the specific 
management courses. To help achieve this objective, survey 
questions 8d and 9 were developed. 
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Question 8d: Overall, I was satisfied with this course. 



Course Avq . 

Basic Project Management 4.03 

Dale Carnegie Effective Speaking 4.30 



The averages here indicate that the participants agreed with 
the statement. However, the participants of the DC course 
agreed somewhat more strongly because there was not one 
instance of a rating below- four. With the reversed scale 
problem corrected, the average for DC would be 4.7. 

Question 9: Should NAC continue to offer this course? 



Course N % Yes 

Basic Project Management (BPM) 38 97 

Dale Carnegie Effective Speaking (DC) 10 100 



This question and question number 8d are very similar. With 
high averages received for question 8d, it was anticipated 
that high percentages would follow for this question. For 
BPM, only one person out of 38 answered this question 
negatively. His complaint was that the course did not teach 
him how to manage a project. All of the respondents of the DC 
course indicated that NAC should continue to offer this 
course . 

3 . Objective Three 

The purpose of this objective is to determine 
if there is a perceived relationship between the management 
training received and changes in job performance. This 
objective is related to questions 4, 5, and 8b in the survey. 
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Question 4 : Describe any non-observable ways you have 

changed as a result of this training, for example, in your 
thinking, level of understanding, attitude , etc. 

Course: Basic Project Management 

N % 



a. Increased understanding 17 45 

b. Better time utilization through 
planning and organization, e.g., 

cut down on overtime 11 29 

c . None 6 16 

d- Misc (Improved attitude, Better 

customer relations) 4 10 



After content analyzing the responses to this question, it was 
determined that 84 percent of the participants reported 
positive non-observable changes whereas only 16 percent of the 
participants reported no non-observable changes. Of those 
that reported gains, 66 percent reported better understanding 
and improved attitude and 34 percent reported improved results 
on the job. Of the six respondents that reported no non- 
observable changes to this question, two were "directed" to 
take the course and (probably as a result) reported negatively 
on nearly every question in the survey. 

Course: Dale Carnegie EFfective Speaking 

N % 



a. More open-minded, improved 
attitude, accepting of other 

viewpoints 6 60 

b . Improved communications /interpersonal 

skills 2 20 

c. Improved self-confidence 2 20 
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All ten participants reported positive non-observable changes 
as a result of the training. Sixty percent of the 
participants felt that the course improved their attitudes or 
that they had become more open-minded. 

Question 5: Describe any behavioral wavs you have changed 
as a result of this training: what do you do differently now 
on your job as a result of having taken this course? 



Course: Basic Project Management 

N %_ 

a. Better organized/ structured 15 39 

b . None 10 26 

c. Better at planning 4 11 

d. Integration of n big picture 1 ’ data, 

more responsive 4 11 

e. Better at customer/peer relations 2 5 

f. Too busy to apply what was learned 1 3 

g. Misc 2 5 



Seventy one percent of the trainees reported positive changes 
as a result of the training they received, whereas only 29 
percent indicated negative or no changes. Five of the ten 
participants that indicated no change, were the same 
individuals that indicated no change for the previous 
question. However, in this case, only one of the participants 
was ’’directed” to take the course. 

Course: Dale Carnegie Effective Speaking 

N % 

a. Better at working w/ people 7 58 

b. Better presentations 2 17 

c. More assertive 2 17 
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d. 



More positive 



1 



8 



For this question, a total of 12 responses was received from 
ten participants, i.e., at least one of the participants made 
more than one response to the question. As can be seen from 
the above responses, all trainees reported a positive 
improvement in their behavior after the training. At 58 
percent, working better with people was the response most 
often mentioned. 

Question 8b: This course met the objectives of 
improving my abilities in this area. 



Course Avg . 

Basic Project Management 3.71 

Dale Carnegie Effective Speaking 4.90 



The average for BPM is the lowest received thus far. However, 
the participants still leaned more toward agreeing with the 
statement than toward being neutral . The high average 
received for the DC course indicates that almost the entire 
sample strongly agreed with the statement. The actual numbers 
for DC were 9 fives and 1 four. This average takes into 
account the reversed scale problem. 

4 . Objective Four 

The purpose of this question is to determine if there 
are any benefits that NAC receives as a result of the 
management training. The data received from survey questions 
6 and 8c provide data relevant to this question. Question 6 
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asks the respondent to consider how his/her observable and 
non-observable changes have benefitted NAC, if at all. 



Q uestion 6: Please consider your answers to questions 4 

and 5 and describe how you think these training outcomes have 
benefitted NAC. 

Course: Basic Project Management 

N % 



a. Working more efficiently 18 47 

b. Improved response time, more 

competition, lower costs 9 24 

c. Better communication/coordination 7 18 

d. Better manager /engineer 6 16 

e . None 6 16 



Sixteen percent of the respondents said that NAC did not 
benefit from the training they had taken. The rest of the 
participants indicated that NAC did, in fact, benefit from the 
training they received. Of the remaining 84 percent, which 
consisted of 40 responses by 36 people, 45 percent said that 
NAC was more efficient as a result of the training received. 
Additionally, 22 percent said that the customers of NAC were 
receiving the benefit in the form of lower costs and improved 
response time. An Interesting note is that there was a least 
one person in each of the first three categories that reported 
no observable benefits in question 5, yet they still felt that 
NAC benefitted from them taking the training. Additionally, 
one respondent who was extremely satisfied with the course 
felt that the course should not be recommended for new 
engineers or technicians. This question received eight more 
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responses than it had participants. This is due to several of 
the participants who gave multiple responses to the question. 
Course: Dale Carnegie Effective Speaking 

N % 

a. Better representation of NAC 

with sponsor 4 40 

b. Working better with other 

employees 7 70 

c . Mi sc 1 10 

Once again , as has been the case with the DC course , a high 
percentage (in this case 100 percent) of the trainees 
indicated that their training has benefitted NAC in a positive 
manner. As indicated above, this course also received more 
responses than participants and this is also the result of 
multiple responses from two participants. 

Question 8c: NAC has benefitted from the management 

training I received. 



Course Avq. 

Basic Project Management 3.66 

Dale Carnegie Effective Speaking 4.80 



This question asks for the same information as number 6. The 
redundancy was built into the question to see if the 
respondents were being consistent with their answers. Upon 
comparing the two questions, it was determined that the 
respondents were being consistent. A somewhat lower positive 
response rate for both questions 6 and 8c was recorded for 
BPM . Whereas, an almost perfect positive response rate was 
recorded for DC. For the DC average, the reversed scale 
problem has already been adjusted. 
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5 . Summary 

a. Basic Project Management 

The data revealed that 100 percent of the engineers 
taking this course had not been formally designated as a 
supervisor. This includes those who were not designated and 
had not supervised other persons during the past year (58%) , 
and those who were not designated but had supervised other 
persons during the past year (42%) . Additionally , 95 percent 

of the engineers who took this course were either G3-lls or 

1 o ~ 

S . 

P^esults of the course content evaluation show that 
4.2 percent of the engineers are having difficulty applying 
what they learned in the class to their job. However, in 
spite of that difficulty, 37 of the 38 respondents indicated 
that NAC should continue to offer this course. This number 
includes five individuals who said that they were "directed” 
to take the course. A possible reason for this disparity is 
that although the material being presented is applicable to 
their jobs, the trainees are not yet in the position where 
they can put to use what they learned. The reason given by 
the one person who thought that the course should not be 
continued at NAC was that it did not teach the trainees how to 
manage a project. However, this one negative statement is 
countered by approximately 37 other positive statements. 



53 



Jb . Dale Carnegie Effective Speaking 

The engineers that took this course had the same 
basic characteristics as the engineers that took Basic Project 
Management. Not only had 100 percent of these engineers not 
been formally designated as supervisors, but they were also 
all GS-lls or 12s. All ten respondents gave positive comments 
and indicated that the course should be continued. Many of 
those same trainees also indicated that this was the best 
training that they had received. 

6 . Remaining Courses 

Since the remainder of the courses had such small 
response rates, they will be looked at concurrently and any 
significant findings will be identified. The first questions 
to be analyzed are questions 8a-d. To review, question 8a 
measured the relevance of the training to the job, 8b was 



designed to measure 


whether or 


not 


the 


training 


improved 


abilities 


in that 


specific area. 


8c 


identified 


if 


NAC 


benef itted 


from the 


training, 


and 


8d 


determined 


if 


the 



employees were satisfied with the course. Table 8 identifies 
the ten remaining courses as well as the average of the 
responses to each question. 

Other than Valuing Diversity, all the other courses have 
averages that indicate positive attitudes toward the training 
received. However, the last five courses only had one 
respondent and are therefore considered to have limited 
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TABLE 8 



AVERAGED RESPONSES TO QUESTION 8 



Course 


# 


Q 8a 


Q 8b 


o 

00 

0 


Q 8d 


Performance Feedback 


6 


4 . 67 


4 .17 


4 . 17 


4 . 00 


Power of Effective 
Listening 


5 


4 . 00 


4.00 


3.80 


3.20 


Train the Trainer 


4 


4 . 75 


4.25 


4.25 


4.25 


Managerial 

Communication Skills 


4 


4 . 50 


4.00 


4 . 50 


4.50 


Supervisory 

Institute 


3 


4 . 67 


4 . 67 


4 . 67 


4 . 67 


Dealing w/ Difficult 
People 


1 


4.00 


4 . 00 


4 . 00 


4 . 00 


Valuing Diversity 


1 


1 . 00 


1 . 00 


1 . 00 


1 . 00 


Assertiveness Skills 


1 


5.00 


4 .00 


4 . 00 


4 . 00 


Teambuilding 


1 


5.00 


4 . 00 


5.00 


4 . 00 


Statistical Process 
Control 


1 


3.00 


4 . 00 


4.00 


5.00 


Combined Average 


27 


4 . 01 


3.81 


3.92 


3.86 



reliability. Overall, as can be seen from the combined 
averages in the last row of Table 8, the respondents felt that 
1) the courses were relevant to their jobs, 2) they improved 
their abilities in that area, 3) NAC benefitted from the 
training, and 4) satisfied with the course. 
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a . Performance Feedback 



The individuals who took this course were people 
who had worked at NAC for more than 11 years , were either a 
GM-13 or 14 , and had been a designated supervisor from 
somewhere between one to ten years. Some of the positive 
gains reported from the course were that they are now better 
at interviewing and that they provide feedback more 
frequently. Of the six respondents, two were required to 
attend and of those two, only one answered negatively to the 
questions. The negative respondent, when asked if NAC should 
continue to offer this course, said no because he felt that 
the course should only be offered to new supervisors. 
However, both respondents were consistently responsible for 
giving the lowest scores aS their answers to question number 
eight . Instead of strongly agreeing that the course was 
relevant to the job, these two respondents only agree with the 
statement. Additionally , they were neutral as to the course 
improving their abilities in that area and to their overall 
satisfaction level . 

b. Power of Effective Listening 

Of the responding sample of participants who have 
taken this course, the majority of the personnel were GS-12s 
and GM-13s who had been working at NAC for approximately 6-10 
years. The averages from the above table indicate a course 
that is only somewhat effective, receiving the lowest combined 
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averages in three of the four questions (excluding Valuing 
Diversity) . The respondents thought the course was quite 
relevant, were fairly positive that it improved their ability, 
felt less confident that NAC benefitted, and felt even less 
satisfied. However, one of the survey respondents was 
required to take the course and it is the responses from that 
individual that brought down the averages. Additionally, all 
of the respondents, except the individual that was required to 
take the course, had positive comments about the training; 
e.g., improved concentration, better communication, and 
improved efficiency . 

c. Train the Trainer 

Four engineers responded to this survey. All of 
their responses to the questions were positive except for one 
comment. One individual indicated that the course did not 
teach instruction methods, the primary purpose for taking the 
course. From the rest of the participants, a sampling of 
their responses are: the course gave them more confidence in 
their teaching skills, it made them more conscious of body 
language, and it benefitted NAC by making the participants 
more organized and more effective as instructors. The 
respondents are GS-12 s and GM-13s that are not formally 
designated supervisors, and the majority of which, three out 
of four, have been employed at NAC for over 20 years. 
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d. RGms.inJ.no Coursos 

Of the seven remaining courses, six consisted of 
positive comments about training. However, due to the small 
number of responses received, no significant conclusions can 
be obtained that would be relevant to the analysis. 
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V. RECOMMENDATIONS AND SUMMARY 
A. RECOMMENDATIONS 

Based on the data analyzed in this study, the following 
recommendations are made: 

1) NAC should continue to systematically collect these 
same kinds of data both immediately after people have taken 
the course as well as 6-12 months later. 

2) NAC should design and implement a training evaluation 
program to allow for a more comprehensive review of the 
courses being taught--if not for all the courses, then at 
least for the courses with high through-put and/or cost. 

3) NAC should conduct a cost benefit study of several of 
the courses evaluated in this study. 

1. Recommendation 1 

This recommendation will give NAC a more accurate 
picture of how representative these data are and if the 
employees truly value these particular courses. This is 
probably the simplest and least expensive method of evaluating 
training but it is also the least effective. Simply measuring 
whether or not an individual liked the training does not 
indicate whether anything was learned. This evaluation would 
measure reaction data, in part, but as with this study, it 
should attempt to go further by assessing perceptions of 
learning, behavior, results, and relevance. By asking these 
questions after a period of time has passed, it will thus 
eliminate the potential positive bias that can result from 
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good feelings in the course. In this way, NAC can find out 
the trainees' perceptions of what really happens when they 
apply what they learn in the course to their jobs. 

2 . Recommendation 2 

Recommendation Number 2 would require more effort but 
will be of more benefit to NAC. The suggestions listed below, 
if followed, should lead NAC to a more effective system for 
evaluating their training program. 

a. At a minimum, NAC should test the participants 
both before and after every course of instruction. This would 
help the training director in determining how much each of the 
participants have learned individually. Additionally, control 
groups should be used whenever possible. This would help in 
determining whether the training was the cause of the increase 
or whether good mentoring or on-the-job experience was 
responsible for the increase. 

b. The next step, after step one is successfully 
implemented, is to measure behavioral changes. This step is 
more difficult. Changes in behavior do not take place over 
night and they do not occur simultaneously for all 
individuals. To measure changes in behavior, the trainees' 
supervisor, with input from the training director, must 
measure behavior before and after the program. Two things 
should be remembered: 1) allow enough time after the training 
for the change to occur, and 2) measure it several times to be 
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sure that the measurement is accurate. As in step one, use a 
control group whenever possible. Many factors can contribute 
to a change in behavior; raises or decreases in salary, 
attitude of the supervisor, or peer pressure from fellow 
employees. Without a control group, the training director 
would not be able to distinguish between changes due to the 
training or changes due to outside influences. 

These guidelines, which were developed by Donald 
Kirkpatrick, were outlined in Chapter II of this study. They 
offer a reliable but often-times difficult means of evaluating 
training. Recommendation Number 2 offers the ideal type of 
evaluation and is the most difficult to implement. As a 
result, most organizations don't do this much evaluation. 
However, just because it may difficult does not mean that the 
evaluation should not be attempted. In the current fiscal 
climate it becomes increasingly important to do what you can 
to demonstrate the value of training. As noted by Jackson and 
Kulp, however, [Ref 18] getting a little bit of data is better 
than getting nothing at all. 

3 . Recommendation 3 

The third recommendation is an attempt to determine 
whether it is worth NAC' s time, money, or effort to continue 
offering several of their courses. The training records 
indicate that seven of the twelve management training courses 
evaluated had fewer than eleven participants take the course 



61 



in the last year and a half. Is it worth NAC' s time and money 
to give a course to such a small portion of their scientists 
and engineers? The only way to know for sure is to conduct a 
cost benefit analysis. 

B . SUMMARY 

The main purpose of this study was to determine whether or 
not the management training program being offered by NAC to 
their scientists and engineers was perceived as beneficial. 
In order to find the answer to this question, four research 
questions were identified: 1) How relevant is management 
training to job requirements according to trainees; 2) How 
satisfied are trainees with specific management courses; 3) 
What is the perceived relationship between management training 
received and changes in job performance; and 4) What are the 
potential benefits of the management training for the NAC 
organization? A survey, with the above questions incorporated 
into it, was developed to help answer those questions. 

The results of the survey indicated that the trainees had 
positive responses to all of the above questions with the 
exception of question number one. This question was answered 
negatively 42 percent of the time for one course only: Basic 
Project Management. All of the other courses for this 
question were answered positively. However, due to the small 
number of surveys returned, the results for the entire study 
must be considered exploratory. 
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APPENDIX A 

NAVAL AVIONICS CENTER ORGANIZATIONAL CHAP.T 
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APPENDIX B 



MANAGEMENT COURSES AVAILABLE TO NAC PERSONNEL 



The following is a list and short description of the 
courses available to NAC managers and management development 
candidates who would like suggestions for their IDP: 



1 . AMA Self-Study Courses 

~ General Management Skills 

• Marketing Management 

• Management and the Computer 

• Manufacturing Management 

• Managerial Finance and Control 

2. Supervisory Development (parts I and II) 

Part I 

• Motivating the Average Employee 

• Improving Performances, Work Habits f and Attendance 

• Maintaining Improvement 

• Taking Disciplinary Action 

• Handling Employee Complaints 

Part II 

• Establishing Goals and Objectives 

• Effective Delegation 

• Interviewing and Selection 

• Giving Bad News 

• Managing Change 

• Time Management 

3 . Supervisory Institute (three day course) 

Day 1 

• Leadership Styles 

• Transition to Supervision 

• Performance Standards/Appraisals 

• Enhancing Employee Motivation 

Day 2 

• Communications: Assertiveness, Praise, and Constructive 

Criticism 
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